A Case Study of Networked Portfolio Assessment System for 

Cooperative Learning 



Abstract 

This study examines the usefulness of a networked portfolio assessment system for cooperative 
learning. The system is with instructional methods that expresses ideas via written report and oral 
presentation, accumulated dialogues through inquiries between peers, critical thinking via 
peer-assessment and self-assessment, and knowledge construction via practical assignment. In this 
pilot study, both grades of peer-assessment and instructor-assessment were used to evaluate the 
students’ performance. Students answered a post questionnaire that measured their attitudes toward the 
system. Forty-six juniors, majoring in information management, enrolled in a course called “Internet 
Technology” and were assigned to twenty-three teams. Each team was assigned to implement a 
web-based system (e.g. an on-line travel system or an on-line bookstore) capable of performing certain 
functions. The quantitative results revealed that the students’ achievements increased significantly 
from first round to second round and that more students were willing to join the learning activities 
similar to this study. 
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Background 

Management of portfolios is a means of accumulating the assignments of students. Such 
assignments reveal the learning processes of a particular student (Liu, Lin & Yuan, 2001). Assessment 
of portfolios through peers and self-reflection enhances a student's critical thinking and motivation to 
learn. It is a good alternative assessment method to evaluate college students (Reeves, 2000). In the 
past, the portfolio-assessment focused on the outcome of individual. The combination of 
portfolio-assessment and cooperative learning under a networked system has seldom been explored. 
For example, Chang (2001) designed a web-based learning portfolio system that enables a student to 
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submit his (or her) assignment, collect his (or her) assignment, evaluate other students’ assignments. 



and browse other students’ assignments. Chang (2001) did not consider the functions for cooperative 
learning (e.g., team-forming, discussion forum for team members). 

Researches in cooperative learning claim that students learn better when they learn together 
(Johnson & Johnson, 1989; Qin, Johnson & Johnson, 1995). Cooper and Mueck (1990) have defined 
cooperative learning as a structured and systematic instructional strategy wherein teams work together 
toward a common goal. The learning activities of cooperative learning should include negotiating a 
common goal with team members, being responsible for team members’ learning as well as one’s own, 
assigning complementary roles and tasks to individuals within each group and cultivating social skills 
for effective cooperative learning (Koschmann, Hall & Miyake, 2002; Slavin, 1995; Springer, Stanne 
& Donovan, 1999). Springer, Stanne, and Donovan (1999) claim that learning activities of cooperative 
learning need a more authentic assessment of higher-order thinking and problem solving. Therefore, 
the combination of portfolio-assessment and cooperative learning should be done to enhance each 
other and to form a brand new cooperative-competitive learning strategy. In such a learning 
environment, a student should cooperate with his (or her) team members. Each team should do their 
best to compete with other teams. The author regards cooperative learning as the most important part 
of the cooperative-competitive learning strategy. Furthermore, the cooperative-competitive learning 
strategy could easily be applied in classroom activities and online learning activities. 

Portfolio-assessment for cooperative learning can be implemented as a networked system. In this 
study, the learning system is composed of three distinct functions: the first sub-system, the second 
sub-system and the third sub-system. 

The first sub-system focuses on the management of portfolios (e.g., students can collect, make 
selections from and reflect on themselves and the team). This function was extended from the 
networked portfolio system (Liu, Lin & Yuan, 2001). 

The second sub-system focuses on peer-assessment and self-assessment (extended from 
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networked peer assessment system, Liu, Chiu, Lin & Yuan, 2001). The third sub-system focuses on 



facilitating cooperative learning (e.g., the function of team-forming for instructor to use and the 
discussion forum for team members). 

Pertinent literature 

Paulson, Paulson, and Mayer (1991) said that a portfolio is a purposeful collection of a student’s 
work that tells the story of the student's progress. It is also a collection of items that reveals different 
aspects of an individual’s growth and development over time. Shores and Cathy (1998) have divided 
the portfolio into three types: 1) Private portfolio 2) Learning Portfolio and 3) Pass-along Portfolio. 

Private Portfolio is one you probably already keep (e.g., photographs of some academic 
activities). Learning Portfolio will encourage richer reflection and communication within your program. 
It is the most fun and the most rewarding to implement. Pass-along Portfolio is a condensed vision of 
the first two. 

Russell and Butcher (1999) discovered a number of advantages after analyzing the use of 
portfolios in educational technology courses. One of them is that portfolios allow each student to 
determine what they want to learn and how they demonstrate their knowledge and skills. Portfolios 
also include a lot of information and artifacts and provide a method for students and instructors to do 
outcome assessment. It also allows students to reflect on their assignments and abilities. 

Russell and Butcher (1999) also indicated the limitations of portfolios. One of them is that 
portfolios require more time compared to other evaluation approaches. The benefits of portfolios are 
also not appreciated and understood by some students. There is also lack of research evidence in the 
value of portfolios. 

System 

This study presents a web-based system. Net worked Portfolio Assessment System for 
Cooperative Learning (hereinafter referred to as NetPASS for CL), which coordinates student learning 
in a manner similar to that of researchers, scientists and practitioners. The author utilized the Windows 
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2000 server to be the operating system, the Internet information system 5.0 to be the web server and 



the SQL server 7.0 to be the database management system. The author has implemented the functions 
of NetPASS for CL (referred to figure 1) by using server-side programs to retrieve and store database 
information. These server-side programs were coded with ASP (active server pages). 

Students can turn in their assignment directly through this online system. Because the assignment 
is in the same format as a HTML (hypertext markup language) file, the assignment can be stored 
directly into the file system, which can be read by reviewers through the web browser. 
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Management of portfolios program 
for each team 



Figure 1 . System architecture of NetPASS for CL (adapted from Liu et al., 2001) 

The System administrator of all classes can create a new class through the administration program. Once the 



class is created, the system establishes a new directory using the ID (e.g., coursel) assigned to this class. The 



pre-defined programs are generated as well. Meanwhile, the new directory comes with some programs (referred to 



figure 1). Thus, each class is independent in that it has its own directory. The instructor of this course can manage 
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the students’ learning activities through the administration program, the news program and the team-forming 



program. Class management includes the student's enrollment information, assessments and team assignments. 
The enrolled students can turn in their team assignment, modify their team assignment, receive and give grades, 
receive and give suggestions as well as view the assignment of other students. When a student has turned in his (or 
her) assignment, this assignment will be given a random number by the system together with an established 
directory. This assignment is therefore stored in the directory. In doing so, each assignment can be recorded 
distinctly before and after each round of modification. 



Procedures 

The students were instructed to do the following procedures: 

• Learn the educational objectives of cooperative learning, portfolio-assessment, peer-assessment 
and self-assessment at the start and study real samples from previous semesters to prepare 
students for later activities. (This step is done in the first semester and lasts about 3 hours) 

• Familiarize one’s self with teaching materials covered in this semester. (Each topic was 
scheduled for 1~2 weeks) 

• Be assigned to a team with one student through the simulated annealing K team-forming 
algorithm for heterogeneous grouping (Liu, 2002). The self-efficacy scale (Lin and Liu, 2001) 
is the input of the above algorithm. 

• Discuss the assignment and the criteria with the instructor to make corrections via NetPASS for 
CL and face-to-face (This step lasts about a week). 

1 . The instructor discusses the criteria to mark an assignment with students in classroom teaching. 

2. Students can discuss the assignment with the instructor informally via NetPASS for CL after 
classroom teaching. 

3. The criteria for the assignment are creativity, feasibility and correctness. The ratings are 
divided into ten categories: from “extremely excellent (TO)” to “extremely poor ( 1)”. 

• The assignment completed by the teams is uploaded into the system (This step lasts about a 
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week). 



1 . Teams complete the assignment by themselves. 

2. Teams were instructed to complete the work and submit it to NetPASS for CL within a week. 

Otherwise they receive no credit on this assignment. 

• The system randomly assigns reviewers (each reviewer grades three to four assignments). 

1. The procedure is automatically done by the system after all teams have uploaded their own 

assignments to NetPASS for CL. 

2. In this study, three assignments were assigned to each reviewer according to past experiences 

(Liu et al., 2001). 

• Reviewers grade and comment their own as well as their peers' assignments (This period lasts 
about a week). 

1. Students assess each assignment per day to alleviate loading. 

2. Students only need to write down their reflections about themselves’ assignment when they assess 
their own assignment. 

3. Students must give grades and comments to peers’ assignments when they do peer-assessment. 

• The system notifies the students of their grades and comments. 

1. NetPASS for CL automatically instructs each student to browse the results of their 
peer-assessment and self-assessment via e-mail (after the review process). 

• Based on the comments coming from other teams, students must make corrections or 
modifications. (This period lasts about a week). 

• The above steps are repeated once or more than once (researchers can select times of repetition 
based on their needs). 

1 . In this study, teams were requested to submit their assignments again. 

2. Each team must present their assignment orally in front of other teams and in front of the 
instructor after the assessment procedure. 
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Data Analysis 



Owing to the limited number of students and groups (which was restricted to twenty-three 
groups and forty-six students from a university of science and technology at northern Taiwan), 
experimental research methods could not be used to empirically confirm the partial effects of 
cooperative learning, portfolio-assessment, peer-assessment and self-assessment. In this study, the 
author measured the instant progress between the first round and the second round using peer 
assessment and instructor assessment. Past studies (Lin, Liu & Yuan, 2001; Tsai, Lin & Yuan, 2002) 
show that there is only a significant difference in a student's progress between the first round and the 
third round. 

The author was interested in the students’ attitudes toward NetPASS for CL from a variety of 
perspectives. One of the students’ attitudes is their willingness to join learning activities via NetPASS 
for CL in the near future. The author was also interested to discover more qualitative feedback from 
students via open-ended questions (after participating in learning activities in this study). 

Research Questions 

Research Question 1: Is peer-assessment an effective evaluation method? Does a significant positive 
correlation (external validity) exist between the instructor and the students over the last two 
rounds of evaluation? 

Research Question 2: Does each team progress instantly? Does a significant difference exist between 
the first round and the second round? 

Research Question 3: Were most students satisfied with each component of NetPASS for CL? 




Results 



Analysis of teams' performance 

Table 1 shows that there are significant, positive correlations between the instructor and the 
students in terms of creativity, feasibility and the correctness (after two rounds of evaluation). 



Table 1: Pearson's correlation between instructor and students 



Criteria 


Pearson’s Correlation 


Creativity 


79 ** 


Feasibility 


7g** 


Correctness 


.89** 


*: p < 0.05, **: p < 0.01 



Table 2 shows that there are significant improvements in the area of creativity, feasibility and 
correctness after two rounds of evaluation. This result agrees with a previous study (Lin, Liu & Yuan, 
2001; Johnson & Johnson, 1989; Qin, Johnson & Johnson, 1995). 



Table 2: Descriptive analysis and t test of round 1 and round 2 



Criteria 


Mean 

(From peers) 


Improvement Mean 

(From instructor) 


Improvement 


Creativity (round 1) 


6.68 


6.57 




Creativity (round 2) 


7.08 


Paired t = 2.96** 7.39 


Paired t = 10.22*** 


Feasibility (round 1) 


7.28 


7.16 




Feasibility (round 2) 


7.79 


7.54 

Paired t = 3.43** 


Paired t = 4.44** 


Correctness (round 1) 


7.06 


7.00 




Correctness (round 2) 


7.69 


7.57 

Paired t = 4.32** 


Paired t = 3.21** 


**: p < 0 . 01 , ***: p < 0.001 









Analysis of students' attitudes toward NetPASS for CL 

• Willingness to join learning activities via NetPASS for CL in the near future: Question 1 shows 
that 65% (chi-square =4.26*) of the students are willing to join learning activities via NetPASS 
for CL in the near future. Feedback from structured interviews indicates that majority of students 
regard learning activities as effective and benefit tremendously from NetPASS for CL. 



• Satisfied with NetPASS for CL in sum : Question 2 shows that 72% (chi-square =8.70**) of 
students are satisfied with NetPASS for CL in sum. 



• Convenient in submitting an assignment: Question 3 shows that 76% (chi-square =12.52***) of 
students agree that submitting an assignment via NetPASS for CL is easy. 



• Satisfied with the user interface of presenting an assignment on the browser: Question 4 also 
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shows that 93% (chi-square =34.78***) of students are satisfied with user interface of presenting 
a report on the browser. 

• Convenience in assessing an assignment: Question 5 shows that 74% (chi-square =10.52***) of 
students agree that assessing peers' assignment via NetPASS for CL is easy. 

• Satisfied with the user interface of presenting reviewers' evaluation on the browser: Question 6 
shows that 85% (chi-square =22.26***) of students are satisfied with the user interface of 
presenting reviewers' evaluation on the browser. 

• Convenience in managing one’s portfolio: Question 7 shows that 83% (chi-square =19.57***) of 
students agree that managing each team’s portfolio via NetPASS for CL is easy. 

• Perceived usefulness of the management of portfolios program: Question 8 shows that 78% 
(chi-square =14.70***) of students agree that management of portfolios program is useful. 

• Convenience of the discussion forum program: Question 9 shows that 87% (chi-square 
=25.13***) of students agree that discussing with their team members via NetPASS for CL is 
easy. 

• Perceived usefulness of the discussion forum program: Question 10 shows that 83% (chi-square 
=19.57***) of the students agree that the discussion forum program is useful. 

• Satisfied with the user interface in presenting course information on the browser: Question 11 
shows that 85% (chi-square =22.26***) of students are satisfied with the user interface in 
presenting reviewers' evaluation on the browser. 

• Satisfied with the team members dispatched via team-forming program: Question 12 shows that 
80% (chi-square =17.04***) of the students are satisfied with the team members dispatched via 
the team-forming program. 
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Discussion and Conclusion 

This study examined the usefulness of NetPASS for CL. Students’ learning is achieved through 
two consecutive assessments and related improvements. In doing so, students hopefully make progress 
in the second round. The author has assigned the same reviewers to grade the same assignment in every 
round. Doing so allows the same reviewers to examine whether or not the assignment has been 
modified according to previous comments. The result shows that students improved each round. 
However, this study did not include a conttol group, thereby leaving some related issues unresolved. 
For example, the team-forming through automatic procedure is better than through each student’s own 
choice. As a whole, the use of the web-based system facilitates collaborative learning and the progress 
of the lead teams’ assignments. 

This study also examined the usability of NetPASS for CL. Sixty five percent of students are 
willing to join learning activities via NetPASS for CL in the near future. The shortages of learning 
activities under NetPASS for CL, as pointed out by another 35% of students, were as follows: 1) More 
peer pressure compared to other courses (38%) 2) Learning procedures are time consuming (31%) 3) 
Some teams tend to give extremely low scores to some assignments (19%) 4) Suggestions from other 
teams are useless for some assignments (6%) 5) Team members have their own ideas and are hard to 
negotiate with (6%). 

Some of these shortages, according to students, may be advantages for instructors. Students have 
revised their own assignments due to greater peer pressure compared to other courses. Of course, the 
learning activities and the team-forming algorithm should be revised in order to increase the 
satisfactory performance of students. 

Second, seventy two percent of students are satisfied with NetPASS for CL. The shortages of the 
system, according to some students, were as follows: 1) Slow downloading time (100%). The speed of 
downloading is related to the outdated network devices. One way to address this problem is to upgrade 
the outdated network devices or indicate the response times of the downloading feature next to the 
hyperlink (Nielsen, 1999). 
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Third, most students believe that the user interface of presenting an assignment on the browser is 



easy to use in submitting an assignment, in assessing peers' assignment, in presenting the reviewers' 
evaluation on the browser, and so forth. 

Finally, this preliminary study also proved the usability of major functions of NetPASS for CL. In 
the next study, the author should include more qualitative analyses on the interactions within a team 
(Koschmann, Hall & Miyake, 2002) and adopt the experimental design to prove more detailed research 
questions. 

Acknowledgments 

The author would like to thank the National Science Council of the Republic of China for 
financially supporting this research under Contract No. NSC 91-251-S159-001. 

References 

Chang, C. C. (2001). Construction and evaluation of a web-based learning portfolio system: An 
electronic assessment tool. Innovations in Education and Teaching International, 38(2), 144-155. 
Johnson, D. W„ & Johnson, R. T. (1989). Cooperative and competition: Theory and research. Edina, 
MN: Interaction Book Company. 

Koschmann, T., Hall, R., & Miyake, N. (Eds.)(2002). CSCL2: Carrying forward the conversation. 
Mahwah, NJ: Lawrence Erlbaum. 

Lin, S. S. J., & Liu, E. Z. F. (2001). College students’ networked peer assessment and learning 
motivation. 2001 Annual Conference of American Psychological Association, USA. 

Lin, S. S. -J., Liu, E. Z. -F., & Yuan, S. -M. (2001). Learning effects of Web-based peer assessment for 
students with various thinking styles. Journal of Computer-Assisted Learning, 17(4), 420-432. 

Liu, E. Z. F. (2002). To propose an algorithm for team forming: Simulated annealing K team-forming 
algorithm for heterogeneous grouping. 2002 Conference of E-Learn, Canada. 



12 




Liu, E. Z. F., Chiu, C. H., Lin, S. S. J., & Yuan, S. M. (2001). Web-based Peer review: An effective 



web-learning strategy with the learner as both adapter and reviewer. IEEE Transactions on 
Education, 44(3), 246-25 1 . 

Liu, E. Z. -F., Lin, S. S. -J., & Yuan, S. -M. (2001). Design of a Networked Portfolio System. British 
Journal of Educational Technology, 32(4), 492-494. 

Nielsen, J. (1999). Designing web usability: The practice of simplicity. New Riders Publishing. 

Paulson, F. L., Paulson, P. R., & Meyer, C. A. (1991). What makes a portfolio a portfolio?. Educational 
Leadership, 48, 60-63. 

Qin, Z., Johnson, D. W., & Johnson, R. W. (1995). Cooperative versus competitive efforts and problem 
solving. Review of Educational Research, 65(2), 129-143. 

Reeves, T. C. (2000). Alternative assessment approaches for online learning environments in higher 
education. Journal of Educational Computing Research, 23(1), 101-111. 

Russell, J. D., & Butcher, C. (1999). Using portfolio in educational technology courses. Journal of 
Technology and Teacher Education, 7(4), 279-289. 

Slavin, R. E. (1995). Cooperative learning: Theory, research, and practice (2 IKl ed.). Boston: Allyn & 
Bacon. 

Shores, E. F., & Cathy, G. (1998). The portfolio book: A step-by-step guide for teachers. Gryphon 
House, Inc. 

Springer, L., Stanne, M. E., & Donovan, S. S. (1999). Effects of small group learning on 
undergraduates in science, mathematics, engineering and technology: A meta-analysis. Review of 
Educational Research, 65(2), 129-143. 



13 




